Real-time visualization of autonomous vehicle behavior in mobile applications

ABSTRACT

Technologies for providing real-time visualizations of a behavior of an autonomous vehicle (AV) associated with a ride request. In some examples, a method for providing real-time visualizations of a behavior of an AV associated with a ride request can include receiving a user request for a ride from an AV, wherein the user request specifies a pick-up location associated with a user; receiving sensor data from one or more sensors associated with the AV; determining, based on the sensor data, a state and context of the AV while the AV is en route to the pick-up location; and presenting, at a display interface, a map depicting one or more visual indicators of the state and context of the AV, the state and context of the AV including a location of the AV and one or more AV operations.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of U.S.application Ser. No. 16/569,488, filed on Sep. 12, 2019, entitledREAL-TIME VISUALIZATION OF AUTONOMOUS VEHICLE BEHAVIOR IN MOBILEAPPLICATIONS, which is expressly incorporated by reference herein in itsentirety.

TECHNICAL FIELD

The present disclosure generally relates to tracking and visualizing astate of an autonomous vehicle.

BACKGROUND

An autonomous vehicle is a motorized vehicle that can navigate without ahuman driver. An example autonomous vehicle can include various sensors,such as a camera sensor, a light detection and ranging (LIDAR) sensor,and a radio detection and ranging (RADAR) sensor, amongst others. Thesensors collect data and measurements that the autonomous vehicle canuse for operations such as navigation. The sensors can provide the dataand measurements to an internal computing system of the autonomousvehicle, which can use the data and measurements to control a mechanicalsystem of the autonomous vehicle, such as a vehicle propulsion system, abraking system, or a steering system. Typically, the sensors are mountedat specific locations on the autonomous vehicles.

Autonomous vehicles can be implemented by companies to provideself-driving car services for the public, such as taxi or ride-hailing(e.g., ride sharing) services. The self-driving car services canincrease transportation options and provide a flexible and convenientway to transport users between locations. To use a self-driving carservice, a user will typically request a ride through an applicationprovided by the self-driving car service. When requesting the ride, theuser can define a specific pick-up and drop-off location, which theself-driving car service can use to identify the route of the user andselect a nearby autonomous vehicle that is able to provide the requestedride to the user. The self-driving car service can then deploy theautonomous vehicle to pick up and transport the user to the drop-offlocation. Upon reaching the drop-off location, the user can disembarkthe vehicle and continue to their final destination.

Unfortunately, the pick-up and drop-off experience of users can beconfusing and unclear. For example, when picking up users, autonomousvehicles do not always pull over at the location the user anticipates.This can be due to various reasons including traffic and environmentconditions, for example. In addition, current ridesharing applicationsoften do not have accurate vehicle location and wait time information.Thus, users can have a difficult time finding their requested autonomousvehicle at pick-up, and are frequently left searching for their ridewhen the estimated time of arrival comes and goes. Moreover, currentridesharing applications do not provide contextual information toexplain why the vehicle may be behaving in a particular way. As aresult, users are often left frustrated when a vehicle is shown as stuckor waiting in an undesirable pick-up location.

BRIEF DESCRIPTION OF THE DRAWINGS

The various advantages and features of the present technology willbecome apparent by reference to specific implementations illustrated inthe appended drawings. A person of ordinary skill in the art willunderstand that these drawings only show some examples of the presenttechnology and would not limit the scope of the present technology tothese examples. Furthermore, the skilled artisan will appreciate theprinciples of the present technology as described and explained withadditional specificity and detail through the use of the accompanyingdrawings in which:

FIG. 1 illustrates an example autonomous vehicle environment including acomputing system in communication with an autonomous vehicle, inaccordance with some examples;

FIGS. 2A and 2B illustrate example interfaces presented on a user'sclient device for tracking an autonomous vehicle after requesting a rideservice, in accordance with some examples;

FIGS. 3A through 3D illustrate example interfaces displaying a state ofan autonomous vehicle while en route to pick up a user at a pick-uplocation, in accordance with some examples;

FIG. 4 illustrates an example interface for presenting a video feed fromone or more camera sensors on an autonomous vehicle, in accordance withsome examples;

FIG. 5 is a diagram illustrating a streaming of sensor data to a clientdevice for presenting map data and autonomous vehicle state and contextinformation, in accordance with some examples;

FIG. 6 illustrates an example method for providing real-time or nearreal-time visualizations of a state and behavior of an autonomousvehicle, in accordance with some examples; and

FIG. 7 illustrates an example computing system architecture forimplementing various aspects of the present technology.

DETAILED DESCRIPTION

Various examples of the present technology are discussed in detailbelow. While specific implementations are discussed, it should beunderstood that this is done for illustration purposes only. A personskilled in the relevant art will recognize that other components andconfigurations may be used without parting from the spirit and scope ofthe present technology. In some instances, well-known structures anddevices are shown in block diagram form in order to facilitatedescribing one or more aspects. Further, it is to be understood thatfunctionality that is described as being carried out by certain systemcomponents may be performed by more or fewer components than shown.

The disclosed technologies address a need in the art for providingintelligent, accurate, and granular vehicle behavior and stateinformation in car service applications. In some aspects, an autonomousvehicle (AV) can implement various sensors and devices to understand itsstate and surrounding environment, and leverage its knowledge of theAV's state and surrounding environment and its more precise locationinformation to provide a rider real-time (or near real-time)visualizations of, and insights into, the AV's location and behavior asthe rider awaits to be picked up by the AV.

For example, the AV can leverage sensors on the vehicle, such as camerasensors, global positioning system (GPS) sensors, radars, lightdetection and ranging (LIDAR) sensors, and inertial sensors, to provideriders a real-time (or near-real time) visualization of the AV'saccurate location and behavior and inform riders about what the AV isexperiencing and doing as it travels to the rider's pick-up location. Insome examples, the AV can provide such location and contextualinformation to an application on a rider's device, which can use theinformation to provide map enhancements such as traffic lightinformation, road conditions, brake light indicators, turn-signalindicators, AV velocity data, steering wheel rotation information, etc.In some examples, the AV can stream to the application on the rider'sdevice a video feed from cameras on the AV. The video feed can depict ascene and/or surrounding environment associated with the AV. Riders canuse such map enhancements and video to understand and track a state,location, and behavior of the AV while the AV is en route or waiting.

FIG. 1 illustrates an example autonomous vehicle environment 100. Theexample autonomous vehicle environment 100 includes an autonomousvehicle 102, a remote computing system 150, and a ridesharingapplication 172 on a computing device 170. The autonomous vehicle 102,remote computing system 150, computing device 170 (including ridesharingapplication 172) can communicate with each other over one or morenetworks, such as a public network (e.g., a public cloud, the Internet,etc.), a private network (e.g., a local area network, a private cloud, avirtual private network, etc.), and/or a hybrid network (e.g., amulti-cloud or hybrid cloud network, etc.).

The autonomous vehicle 102 can navigate about roadways without a humandriver based on sensor signals generated by sensors 104 on theautonomous vehicle 102. The sensors 104 on the autonomous vehicle 102can include one or more types of sensors and can be arranged about theautonomous vehicle 102. For example, the sensors 104 can include,without limitation, one or more: inertial measuring units (IMUs), imagesensors (e.g., visible light image sensors, infrared image sensors,video camera sensors, etc.), light emitting sensors (e.g., lasers, lightdetection and ranging sensors (LIDARs), etc.), global positioning system(GPS) devices, radars, sonars, accelerometers, gyroscopes,magnetometers, motion detection sensors, light detectors, audio sensors(e.g., microphones), seat occupancy sensors, seat belt tension orengagement sensors, braking sensors, ultrasonic sensors, positionsensors, steering angle sensors, steering wheel rotation sensors, turnsignal sensors, parking sensors, gear position sensors, brake lightsensors, odometers, speedometers, proximity sensors, temperaturesensors, etc.

In some implementations, the sensors 104 can include one or more radarssensors 106A, one or more image sensors 106B (e.g., camera sensors,visible light sensors, infrared light sensors, etc.), and one or moreLIDAR sensors 106N. Other implementations can include any other numberand types of sensors.

The autonomous vehicle 102 can include one or more display devices 108for presenting information, such as maps, messages, and interfaces, topassengers in the autonomous vehicle 102. The one or more displaydevices 108 can be mounted on one or more locations in the autonomousvehicle 102. For example, the one or more display devices 108 can bemounted on one or more seats or headrests in the autonomous vehicle 102,a dashboard in the autonomous vehicle 102, one or more inner sides ordoor panels on the autonomous vehicle 102, a roof of the autonomousvehicle 102, and/or any other interior location of the autonomousvehicle 102. The one or more display devices 108 can include, forexample and without limitation, a screen, a television, a projectingdevice, and/or any other suitable display device for rendering graphicalinformation.

Moreover, the autonomous vehicle 102 can include several mechanicalsystems that are used to effectuate motion of the autonomous vehicle102. For instance, the mechanical systems can include, but are notlimited to, a vehicle propulsion system 130, a braking system 132, and asteering system 134. The vehicle propulsion system 130 can include anelectric motor, an internal combustion engine, or both. The brakingsystem 132 can include an engine brake, brake pads, actuators, and/orany other suitable componentry configured to assist in decelerating theautonomous vehicle 102. The steering system 134 includes suitablecomponentry configured to control the direction of movement of theautonomous vehicle 102 during navigation.

The autonomous vehicle 102 can include a safety system 136. The safetysystem 136 can include lights and signal indicators, a parking brake,airbags, etc. The autonomous vehicle 102 can also include a cabin system138, which can include cabin temperature control systems, in-cabinentertainment systems, display devices, light-emitting devices, audiosystems, etc.

The autonomous vehicle 102 can include an internal computing system 110in communication with the sensors 104, the display device(s) 108, andthe systems 130, 132, 134, 136, and 138. The internal computing system110 can include one or more processors and at least one memory forstoring instructions executable by the one or more processors. Thecomputer-executable instructions can make up one or more services forcontrolling the autonomous vehicle 102, communicating with remotecomputing system 150, receiving inputs from passengers or humanco-pilots, logging metrics regarding data collected by the sensors 104and human co-pilots, etc.

The internal computing system 110 can include a control service 112configured to control operation of the vehicle propulsion system 130,the braking system 132, the steering system 134, the safety system 136,and the cabin system 138. The control service 112 can receive sensorsignals from the sensors 104 can communicate with other services of theinternal computing system 110 to effectuate operation of the autonomousvehicle 102. In some examples, control service 112 may carry outoperations in concert with one or more other systems of autonomousvehicle 102.

The internal computing system 110 can also include a constraint service114 to facilitate safe propulsion of the autonomous vehicle 102. Theconstraint service 116 includes instructions for activating a constraintbased on a rule-based restriction upon operation of the autonomousvehicle 102. For example, the constraint may be a restriction onnavigation that is activated in accordance with protocols configured toavoid occupying the same space as other objects, abide by traffic laws,circumvent avoidance areas, etc. In some examples, the constraintservice 114 can be part of the control service 112.

The internal computing system 110 can also include a communicationservice 116. The communication service 116 can include software and/orhardware elements for transmitting and receiving signals to and from theremote computing system 150. The communication service 116 can beconfigured to transmit information wirelessly over a network, forexample, through an antenna array or interface system that providescellular (long-term evolution (LTE), 3^(rd) Generation (3G), 5^(th)Generation (5G), etc.) communication, Bluetooth communication,near-field communication, and/or any other suitable type of wirelesscommunication.

In some examples, one or more services of the internal computing system110 are configured to send and receive communications to remotecomputing system 150 for reporting data for training and evaluatingmachine learning algorithms, requesting assistance from remote computingsystem 150 or a human operator via remote computing system 150, softwareservice updates, ridesharing pickup and drop off instructions, etc.

The internal computing system 110 can also include a latency service118. The latency service 118 can utilize timestamps on communications toand from the remote computing system 150 to determine if a communicationhas been received from the remote computing system 150 in time to beuseful. For example, when a service of the internal computing system 110requests feedback from remote computing system 150 on a time-sensitiveprocess, the latency service 118 can determine if a response was timelyreceived from remote computing system 150, as information can quicklybecome too stale to be actionable. When the latency service 118determines that a response has not been received within a thresholdperiod of time, the latency service 118 can enable other systems ofautonomous vehicle 102 or a passenger to make decisions or provideneeded feedback.

The internal computing system 110 can also include a user interfaceservice 120 that can communicate with cabin system 138 to provideinformation or receive information to a human co-pilot or passenger. Insome examples, a human co-pilot or passenger can be asked or requestedto evaluate and override a constraint from constraint service 114. Inother examples, the human co-pilot or passenger may wish to provide aninstruction to, or receive an instruction from, the autonomous vehicle102 regarding destinations, requested routes, drop-off locations,wayfinding tasks, or other requested operations.

As described above, the remote computing system 150 can be configured tosend and receive signals to and from the autonomous vehicle 102. Thesignals can include, for example and without limitation, data reportedfor training and evaluating services such as machine learning services,data for requesting assistance from remote computing system 150 or ahuman operator, software service updates, rideshare pickup and drop offinstructions, etc.

The remote computing system 150 can include an analysis service 152configured to receive data from autonomous vehicle 102 and analyze thedata to train or evaluate machine learning algorithms for operating theautonomous vehicle 102. The analysis service 152 can also performanalysis pertaining to data associated with one or more errors orconstraints reported by autonomous vehicle 102.

The remote computing system 150 can also include a user interfaceservice 154 configured to present metrics, video, images, soundsreported from the autonomous vehicle 102 to an operator of remotecomputing system 150, maps, routes, navigation data, notifications, userdata, vehicle data, software data, and/or any other content. Userinterface service 154 can receive, from an operator, input instructionsfor the autonomous vehicle 102.

The remote computing system 150 can also include an instruction service156 for sending instructions regarding the operation of the autonomousvehicle 102. For example, in response to an output of the analysisservice 152 or user interface service 154, instructions service 156 canprepare instructions to one or more services of the autonomous vehicle102 or a co-pilot or passenger of the autonomous vehicle 102.

The remote computing system 150 can also include a rideshare service 158configured to interact with ridesharing applications 172 operating oncomputing device 170. Computing device 170 can include, for example andwithout limitation, a tablet computer, a laptop computer, a smartphone,a head-mounted display (HMD), a gaming system, a server, a smart device,a laptop computer, a smart wearable (e.g., smart glasses, smart watch,etc.), a mobile computing device, and/or any other computing device. Insome cases, the computing device 170 can be a mobile computing device ofa user or passenger/rider. Moreover, in some cases, the remote computingsystem 150 and the autonomous vehicle 102 can also communicate andinteract with other computing devices hosting instances of theridesharing application 172 and the delivery service 174. For example,the remote computing system 150 and the autonomous vehicle 102 can alsocommunicate and interact with other computing devices associated withone or more passengers.

The rideshare service 158 can receive requests from passengerridesharing application 172, such as user requests to be picked up ordropped off, and can dispatch autonomous vehicle 102 for a requestedtrip. The rideshare service 158 can also act as an intermediary betweenthe ridesharing application 172 and the autonomous vehicle 102. Forexample, rideshare service 158 can receive from a passenger instructionsfor the autonomous vehicle 102, such as instructions to go around anobstacle, change routes, select a drop-off location and/or pick-uplocation, honk the horn, open doors in the autonomous vehicle 102, etc.The rideshare service 158 can provide such instructions to theautonomous vehicle 102 as requested.

The remote computing system 150 can also include a package service 162configured to interact with the computing device 170, the ridesharingapplication 172 and/or a delivery service 174 of the ridesharingapplication 172. A user operating the ridesharing application 172 caninteract with the delivery service 174 to specify information regardinga package to be delivered using the autonomous vehicle 102. Thespecified information can include, for example and without limitation,package dimensions, a package weight, a destination address, deliveryinstructions (e.g., a delivery time, a delivery note, a deliveryconstraint, etc.), and so forth.

The package service 162 can interact with the delivery service 174 toprovide a package identifier to the user for package labeling andtracking. Package delivery service 174 can also inform a user of whereto bring their labeled package for drop off. In some examples, a usercan request the autonomous vehicle 102 come to a specific location, suchas the user's location, to pick up the package. While delivery service174 has been shown as part of the ridesharing application 172, it willbe appreciated by those of ordinary skill in the art that deliveryservice 174 can be its own separate application.

One example beneficial aspect of utilizing autonomous vehicle 102 forboth ridesharing and package delivery is increased utilization of theautonomous vehicle 102. Instruction service 156 can continuously keepthe autonomous vehicle 102 engaged in a productive itinerary betweenrideshare trips by filling what otherwise would have been idle time withproductive package delivery trips.

FIG. 2A illustrates an example interface 200 presented on a user'sclient device for tracking the autonomous vehicle 102 after requesting aride service (e.g., 158). The client device can be, for example, amobile device such as a mobile phone, a tablet computer, a smartwearable, etc. In some examples, the client device can be the computingdevice 170 shown in FIG. 1 . For explanation purposes, the client devicehereinafter will be discussed in terms of (e.g., with reference to) thecomputing device 170 shown in FIG. 1 .

The interface 200 can be presented on the client device 170 via anapplication such as the ridesharing application 172 shown in FIG. 1 .Moreover, in some examples, some or all of the information presented inthe interface 200 can be based on sensor data (e.g., GPS data,image/video data, radar measurements, LIDAR measurements, inertialmeasurements, triangulation data, braking sensor data, ultrasonic sensordata, position sensor data, steering angle sensor data, steering wheelrotation sensor data, turn signal sensor data, parking sensor data, gearposition sensor data, brake light sensor data, odometer data,speedometer data, accelerometer data, proximity sensor data, temperaturemeasurements, light sensor data, etc.) provided by the autonomousvehicle 102 (e.g., via the internal computing system 110 and/or theremote computing system 150). In some cases, some of the informationpresented in the interface 200 can also be based on sensor data (e.g.,GPS data, triangulation data, image/video data, inertial measurements,etc.) from one or more sensors on the client device 170.

In the example shown in FIG. 2A, the interface 200 can include a map 202of the area associated with the autonomous vehicle 102 and the userrequesting a ride service from the autonomous vehicle 102. The map 202can display a current location 204 of the autonomous vehicle 102 and apath 208 from the current location 204 of the autonomous vehicle 102 toa pick-up location 206 set for the user/trip. In some cases, the map 202can also display tracking information such as, for example, an estimatedtime of arrival 210 to the pick-up location 206, traffic information,route information, etc.

In some implementations, the interface 200 can also include aninformation section 212 for providing information about the trip, suchas vehicle information, traffic information, trip conditions,environment conditions, a state of the autonomous vehicle 102 (e.g.,speed, position, trajectory, vehicle conditions, operation state, etc.),a behavior of the autonomous vehicle 102, road conditions, trip alerts,navigation information, user instructions, etc. For example, theinformation section 212 can display a notification 214 indicating anestimated time of arrival of the autonomous vehicle 102.

In some examples, the autonomous vehicle 102 can provide (e.g., via theinternal computing system 110 and/or the remote computing system 150)updated sensor data to the client device 170 as the autonomous vehicle102 travels from the current location 204 to the pick-up location 206.For example, the autonomous vehicle 102 can provide updated sensor dataperiodically and/or as any changes occur in the location/position of theautonomous vehicle 102, the state of the autonomous vehicle 102, thebehavior and/or operation of the autonomous vehicle 102, roadconditions, vehicle conditions, environment conditions, trip conditions,etc.

The interface 200 can update the map 202 and/or information presented inthe information section 212 based on the updated sensor data provided bythe autonomous vehicle 102. For example, with reference to FIG. 2B, asthe location of the autonomous vehicle 102 (and/or the relativelocation/distance between the client device 170 and the autonomousvehicle 102) changes, the interface 200 can update the map 202 and/orthe information section 212 to present more current information. Toillustrate, the map 202 can present an updated location 220 of theautonomous vehicle 102, an updated path 222 from the updated location220 of the autonomous vehicle 102 to the pick-up location 206, and anindicator 224 of an updated estimated time of arrival of the autonomousvehicle 102.

In some examples, the interface 200 can also update the map 202 todepict a state of the autonomous vehicle 102, an operation and/orbehavior of the autonomous vehicle 102, environment conditions, etc. Toillustrate, the map 202 can display a visual indication 228 of a state(e.g., activated, inactivated) of the brake lights on the autonomousvehicle 102 and a traffic light indicator 226 that visually depicts astate of a traffic light (e.g., green light, yellow light, red light)approached by the autonomous vehicle 102. In the example shown in FIG.2B, the map 202 includes a visual indication 228 of an activated stateof the brake lights, which indicates that the autonomous vehicle 102 hasimplemented the brakes, and a traffic light indicator 226 showing thatthe autonomous vehicle 102 has approached a red traffic light.

Moreover, in some examples, the interface 200 can update the informationsection 212 to present new or updated information. For example, theinterface 200 can present notifications 214 indicating an estimated timeof arrival of the autonomous vehicle 102 and informing the user that theautonomous vehicle 102 is stopped at a red stop light but has a clearpath ahead. In some cases, if there are obstructions along the path 222of the autonomous vehicle 102 or any other conditions relevant to theuser/trip, the interface 200 can similarly update the informationsection 212 to display an indication/description of such obstructionsand conditions.

FIG. 3A illustrates interface 300 displaying a state of the autonomousvehicle 102 while en route to pick up a user at a pick-up location 304.The interface 300 can be displayed on the client device 170 via anapplication such as ridesharing application 172. Moreover, the state ofthe autonomous vehicle 102 can be determined and displayed based on datafrom sensors 104 on the autonomous vehicle 102, which can be provided bythe autonomous vehicle 102 (e.g., via the internal computing system 110and/or the remote computing system 150) to the client device 170.

In this example, the interface 300 includes a map 302, which depicts theautonomous vehicle 102 at a current location 306 and a path 308 of theautonomous vehicle 102 from the current location 306 to a pick-uplocation 304 for the user. The map 302 also depicts an obstruction 310in the path 308 of the autonomous vehicle 102. The obstruction 310 canindicate to the user that the autonomous vehicle 102 is stopped or willlikely have to come to a stop; that the autonomous vehicle 102 may bedelayed; and/or a reason why the autonomous vehicle 102 is stopped, willlikely have to come to a stop, or may be delayed.

The obstruction 310 can be detected by the autonomous vehicle 102 basedon data from sensors 104 on the autonomous vehicle 102. For example, theobstruction 310 can be detected based on an image/video captured by oneor more image sensors on the autonomous vehicle 102, radar returnsobtained using one or more radars on the autonomous vehicle 102, LIDARdata or a LIDAR point cloud obtained from one or more LIDARs onautonomous vehicle 102, GPS data, motion sensor data, measurements froma structured-light 3D (three-dimensional) scanner on the autonomousvehicle 102, etc. The autonomous vehicle 102 can send to the clientdevice 170 the sensor data and/or information about the obstruction 310detected based on the sensor data, such as a description of theobstruction 310 (e.g., a location, a depiction, a configuration, anexplanation, a report, characteristics, etc.). The client device 170 canuse the sensor data and/or information from the autonomous vehicle 102to display the obstruction 310 on the map and present any otherpertinent information at the interface 300.

The obstruction 310 can be any type of obstruction that may be relevantto the user and/or the autonomous vehicle 102. For example, theobstruction 310 can be any type of obstruction that may impact theamount of time it will take the autonomous vehicle 102 to reach thepick-up location 304 from the current location 306 and/or the currenttime, the path that the autonomous vehicle 102 will or should take fromthe current location 306 to the pick-up location 304, or how (or if) theautonomous vehicle 102 can/will access the pick-up location 304.Non-limiting examples of obstructions can include road closures,accidents, obstacles (e.g., stalled vehicle, crossing animal, flooding,pot holes, objects, people, road hazards, emergency personnel, etc.),events (e.g., street/road construction, street fairs, concerts,emergencies, sports events, etc.), traffic, etc.

The interface 300 can also include an information section 312 forpresenting state information 314 (e.g., vehicle state, environmentstate, road conditions, surrounding conditions, etc.). The informationpresented in the information section 312 can include text information(e.g., messages, alerts, notifications, etc.), visual information (e.g.,images, icons, animations, video, color, graphical objects, graphicalvisualizations, etc.), interface elements, etc.

In the example shown in FIG. 3A, the state information 314 presented onthe information section 312 include a visual odometer depicting acurrent speed of the autonomous vehicle 102, a status message indicatingthat the autonomous vehicle 102 is currently stopped, an alertindicating that traffic is stopped because of an accident that iscurrently blocking traffic, a temperature of the interior cabin (e.g.,138) of the autonomous vehicle 102, and a notification of an estimatedtime of arrival of the autonomous vehicle 102.

In other examples, the information section 312 can present otherinformation such as, for example and without limitation, a distancetraveled by the autonomous vehicle 102, a remaining distance from thecurrent location 306 of the autonomous vehicle 102 to the pick-uplocation 304, what gear the autonomous vehicle 102 is currently in, adescription or depiction of the obstruction 310, an indication of otherstate and/or operating parameters of the autonomous vehicle 102 (e.g., astate of any turn-signals or blinkers, a braking state, a state of thebrake lights, a steering wheel rotation, a vehicle battery status, asteering angle, etc.), a description or indication of the environmentaround the autonomous vehicle 102, information alerts, updates, etc.

With reference to FIG. 3B, the interface 300 can change or updatecontent presented in the interface 300 to reflect changes in state orcircumstances (e.g., changes in the vehicle's location, changes in theenvironment, changes in the vehicle's behavior, changes in the vehicle'soperating state, changes in the vehicle's route to the pick-up location,changes in the vehicle's estimated time of arrival, etc.) and/or provideany other information. The interface 300 can change or update suchcontent based on data provided by the autonomous vehicle 102 (e.g., viathe internal computing system 110 and/or the remote computing system150), such as sensor data as previously explained.

In the example shown in FIG. 3B, the map 302 has been updated to reflectan updated location 320 of the autonomous vehicle 102 and an updatedpath 322 from the updated location 320 of the autonomous vehicle 102 tothe pick-up location 304. The map 302 has also been updated to depictthe autonomous vehicle 102 approaching a red traffic light 324 with thebrake lights 326 on the autonomous vehicle 102 turned on. Thevisualization on the map 302 of the autonomous vehicle 102 approachingthe red traffic light 324 with the brake lights 326 on can indicate tothe user that the autonomous vehicle 102 is coming to a stop for a redtraffic light so the user can have a better understanding of why theautonomous vehicle 102 appears to have stopped (or in the process ofstopping) and any consequent delays. Thus, the user can better track theautonomous vehicle 102 and obtain a better understanding of the currentbehavior/operation of the autonomous vehicle 102.

In addition, the information section 312 has been updated to includeinformation 328 about the current state or circumstances associated withthe autonomous vehicle 102. In this example, the information 328includes a visualization of an odometer indicating a current speed ofthe autonomous vehicle 102, an updated status of the autonomous vehicle102 (e.g., stopping), a vehicle battery status, a current vehicletemperature, and an updated notification indicating an estimated time ofarrival calculated at least partially based on the current state orcircumstances associated with the autonomous vehicle 102.

The interface 300 can continue to change or update content presented inthe interface 300 when it receives new or updated data from theautonomous vehicle 102. Such changes or updates can reflect any changesin state or circumstances and/or provide any other information, aspreviously explained. For example, with reference to FIG. 3C, the clientdevice 170 can receive new or updated data from the autonomous vehicle102 and update the map 302 in the interface 300 to reflect a currentlocation 330 of the autonomous vehicle 102 and a remaining path 332 fromthe current location 330 of the autonomous vehicle 102 to the pick-uplocation 304.

In this example, the autonomous vehicle 102 is also depicted in the map302 with a left-turn signal 334 turned on (e.g., blinking or flashing),indicating that the autonomous vehicle 102 is turning (or preparing toturn) left. The visualization in the map 302 of the left-turn signal 334can be based on data (e.g., sensor data) received from the autonomousvehicle 102 indicating that the left-turn signal of the autonomousvehicle 102 has been turned on or activated. Such visualization andinformation can inform the user of the current state/trajectory of theautonomous vehicle 102.

Moreover, the information 336 presented in the information section 312of the interface 300 has been updated to include an indication (e.g., avisualization of an odometer) of a current speed of the autonomousvehicle 102, a notification of a current status of the autonomousvehicle 102 (e.g., turning left, steering wheel turning), and a currentindication of the estimated time of arrival of the autonomous vehicle102.

The interface 300 can provide any other pertinent information that caninform the user of the state and behavior of the autonomous vehicle 102.For example, with reference to FIG. 3D, the map 302 in the interface 300has been updated (e.g., based on data received from the autonomousvehicle 102) to depict the autonomous vehicle 102 in its current state,which in this example shows the autonomous vehicle 102 performing au-turn after passing the pick-up location 304. The map 302 also includesa visual indicator 340 of a u-turn action. If the visualization of theautonomous vehicle 102 performing the u-turn does not clearly convey tothe user what exact maneuver is being performed by the autonomousvehicle 102, the visual indicator 340 can provide additionalinformation/context to help the user understand that the autonomousvehicle 102 is performing a u-turn.

In addition, FIG. 3D shows updated information 342 presented in theinformation section 312 of the interface 300 to provide additionalinformation/context to the user to help the user understand the currentstate and circumstances associated with the autonomous vehicle 102. Inthis example, the updated information 342 includes a message statingthat the autonomous vehicle 102 is performing a u-turn; an indication ofa steering wheel rotation, which can further inform the user of thecurrent operation of the autonomous vehicle 102 (e.g., the u-turn inprocess or planned); and a status indicator which provides that theautonomous vehicle 102 is in a first gear and moving at 4 miles perhour. The gear and speed of the autonomous vehicle 102 in this examplecan again provide the user further state and context information forunderstand the current and planned operation/behavior of the autonomousvehicle 102.

In some examples, in addition to providing current state and behaviorinformation of the autonomous vehicle 102, the interface 300 can alsoprovide future planned, estimated, or predicted state and behaviorinformation. For example, if the autonomous vehicle 102 switches lanesand slows down prior to reaching an exit along a planned route, theinterface 300 can present (e.g., via the map 302 and/or informationsection 312) visual indicators and/or information indicating that theautonomous vehicle 102 plans to, or is estimated/predicted to, turn intoan exit ramp at an upcoming exit. The data used by the interface 300 toprovide such information can be provided by the autonomous vehicle 102(e.g., via the internal computing system 110 and/or the remote computingsystem 150), as previously explained.

For example, the internal computing system 110 on the autonomous vehicle102 can analyze a current location of the autonomous vehicle 102, acurrent velocity of the autonomous vehicle 102, a currentdirection/trajectory of the autonomous vehicle 102, a planned route, arelative direction to the pick-up location and/or any sensed oridentified operations at the autonomous vehicle 102 (e.g., braking,slowing down, speeding up, changing steering angle, changing steeringwheel rotation angle, activating a turning signal, etc.) to determine aplanned or predicted operation (e.g., turning into the exit ramp) of theautonomous vehicle 102. The internal computing system 110 can then sendsuch information to the client device 170, which can optionally updatethe interface 300 to provide one or more indications of the planned orpredicted operation of the autonomous vehicle 102.

FIG. 4 illustrates an example interface 400 for presenting a video feed402 from one or more camera sensors (e.g., 104) on the autonomousvehicle 102. The interface 400 can be rendered on the client device 170via an application such as ridesharing application 172. The autonomousvehicle 102 can stream (e.g., via the internal computing system 110) thevideo feed 402 to the client device 170, which can then present thevideo feed 402 on the interface 400.

In some cases, the one or more camera sensors can be mounted on anexterior/outside of the autonomous vehicle 102 to capture an environmentaround the autonomous vehicle 102. The one or more camera sensors cancapture video of the environment around the autonomous vehicle 102 asthe autonomous vehicle 102 is en route and/or waiting for the user atthe pick-up location or a parked location. The autonomous vehicle 102can obtain the video captured by the one or more camera sensors andstream some or all of the video to the client device 170 forpresentation on the interface 400.

The video feed 402 presented on the interface 400 can allow a user tosee the environment around the autonomous vehicle 102 and obtain insightinto any conditions around the autonomous vehicle 102. This can help theuser understand not only the state or status of the autonomous vehicle102 but also any reasons why the autonomous vehicle 102 may be delayedor deviating from a planned route to the pick-up location. In somecases, the video feed 402 can also help the user understand what theautonomous vehicle 102 is doing as it travels to the pick-up location orwaits for the user to find and/or board the vehicle.

In some cases, the client device 170 can display on the interface 400 a360 degree view of the environment/scene around the autonomous vehicle102 by stitching together camera footage from multiple camera sensors.For example, the client device 170 (or the internal computing system110) can stitch together footage from front, back, and side-facingcamera sensors to generate a 360 degree view of the environment/scenearound the autonomous vehicle 102. In some cases, the client device 170can also display a third-person view of the autonomous vehicle 102 byoverlaying the video to create an impression of the autonomous vehicle102 driving along as the video progresses.

The interface 400 can also include an information section 404 displayingother content such as notifications, alerts, messages, eventdescriptions, state information, instructions, and/or any otherinformation. For example, the information section 404 can displaymessages indicating a current status of the autonomous vehicle 102 andan estimated time of arrival of the autonomous vehicle 102. As anotherexample, the information section 404 can display information correlatedto the content being displayed in the video feed 402. To illustrate, theinformation section 404 can display a message informing the user thatthe autonomous vehicle 102 will park and wait behind an objectidentified in the video feed 402, such as another vehicle or a building.

FIG. 5 is a diagram illustrating a streaming of sensor data 502 to theclient device 170 for presenting map data and AV state and contextinformation as described herein. For example, the client device 170 canuse the sensor data 502 to present any of the information in theinterfaces 200, 300 and/or 400 shown in FIGS. 2A-B, 3A-D, and 4,respectively.

The sensor data 502 can include data and measurements captured by thesensors 104 on the autonomous vehicle 102. For example, the sensor data502 can include or reflect measurements or sensed data from one or moreimage sensors on the autonomous vehicle 102 (e.g., images, video, etc.),one or more LIDARs on the autonomous vehicle 102 (e.g., LIDAR data,LIDAR point cloud information, etc.), one or more radars on theautonomous vehicle 102 (e.g., radar returns, radar return information,etc.), one or more braking sensors on autonomous vehicle 102, one ormore brake light sensors on the autonomous vehicle 102, one or more turnsignal sensors on the autonomous vehicle 102, one or more speedometerson the autonomous vehicle 102, one or more steering wheel rotationsensors on the autonomous vehicle 102, one or more gear sensors on theautonomous vehicle 102, and/or any other sensors as described herein.

In some cases, the sensor data 502 can include raw sensor measurementsand/or information generated based on raw sensor measurements. Forexample, the sensor data 502 can include raw vehicle accelerationmeasurements and/or information describing an vehicle accelerationmeasurements (e.g., data indicating that the autonomous vehicle 102 hasaccelerated by 5 miles per hour (MPH) and/or that the autonomous vehicle102 has increased its speed to 30 MPH). As another example, the sensordata 502 can include raw location/position measurements and/or trackestimates generated by fusing raw measurements from various sensors andpredicting a track of the autonomous vehicle 102 based on the fusedsensor measurements. As yet another example, the sensor data 502 caninclude raw camera sensor footage or processed footage generated basedon raw camera sensor footage using one or more image/video processingtechniques.

In some cases, the client device 170 can receive some or all of thesensor data 502 from the internal computing system 110 on the autonomousvehicle 102. In other cases, the client device 170 can receive some orall of the sensor data 502 from the remote computing system 150associated with the autonomous vehicle 102. For example, the internalcomputing system 110 on the autonomous vehicle 102 can provide thesensor data 502 to the remote computing system 150, which can thenprovide the sensor data 502 (e.g., as is, after being processed, and/orafter being supplemented with other data) to the client device 170. Inyet other cases, the client device 170 can receive a portion of thesensor data 502 from the internal computing system 110 on the autonomousvehicle 102 and another portion of the sensor data 502 from the remotecomputing system 150.

In some cases, the internal computing system 110 on the autonomousvehicle 102 or the remote computing system 150 can process the sensordata 502 prior to providing it to the client device 170. For example, insome cases, prior to providing the sensor data 502 to the client device170, the internal computing system 110 or the remote computing system150 can first verify (e.g., check for accuracy, check for relevance,compare with other measurements, curate, etc.) some or all of the sensordata 502. As another example, prior to providing the sensor data 502, insome cases, the internal computing system 110 on the autonomous vehicle102 or the remote computing system 150 can perform time-event mapping orsynchronization to ensure that data provided to the client device 170 isnot outdated and synchronized with any events, state, or conditionsdescribed/represented by such data.

Moreover, in some cases, the internal computing system 110 on theautonomous vehicle 102 or the remote computing system 150 can calculateand/or vary the time intervals in which sensor data is sent to theclient device 170. For example, the internal computing system 110 on theautonomous vehicle 102 or the remote computing system 150 can sendsensor data as obtained, at predefined intervals, or at variableintervals. In some cases, the internal computing system 110 on theautonomous vehicle 102 or the remote computing system 150 can sendcertain type of sensor data at variable intervals which can depend onthe lifespan/relevance of the data and/or the network/resourcebandwidth.

For example, sensor data describing events that are short-lived or fastpaced can be sent at more frequent intervals to ensure such events areadequately conveyed to the client device 170 and/or that such sensordata is sufficiently relevant or synchronized in time with such events.In another example, sensor data describing such short-lived orfast-paced events may not be sent to the client device 170 in some orall instances if such data is estimated to be outdated by the time itreaches the client device 170.

In some cases, prior to providing the sensor data 502 to the clientdevice 170, the internal computing system 110 or the remote computingsystem 150 can filter out any portions of the sensor data 502. Forexample, the internal computing system 110 or the remote computingsystem 150 can filter out portions of the sensor data 502 that areoutdated, inaccurate, or not needed by the client device 170. In otherexamples, prior to providing the sensor data 502 to the client device170, the internal computing system 110 or the remote computing system150 can supplement the data with other data from other sources.

For example, the internal computing system 110 can send the sensor data502 to the remote computing system 150, where the sensor data 502includes information about the environment around the autonomousvehicle. The remote computing system 150 can receive such sensor data502 and supplement it with other information about the environmentobtained from one or more other sources, such as sensors on otherautonomous vehicles in the fleet, the Internet, government sources, newssources, a third-party map/navigation application service, a geographicinformation system, a data repository associated with the remotecomputing system 150, etc.

In some cases, the client device 170 can receive the sensor data 502 andprocess the sensor data 502 to provide the AV state, behavior, andcontext information described herein. For example, the client device 170can receive the sensor data 502 and use the sensor data 502 to rendersuch information on a display interface such as interface 200, 300, or400 shown in FIGS. 2A-B, 3A-D, and 4.

Having disclosed some example system components and concepts, thedisclosure now turns to FIG. 6 , which illustrates an example method 600for providing real-time (or near real-time) visualizations of a stateand behavior of an autonomous vehicle (102). The steps outlined hereinare examples and can be implemented in any combination thereof,including combinations that exclude, add, or modify certain steps.

At step 602, the method 600 can include receiving (e.g., at the clientdevice 170, the remote computing system 150, and/or the internalcomputing system 110) a user request for a ride from an autonomousvehicle (102). The user request can specify a pick-up location for therequested trip. In some examples, the user request can also specifyother information such as a pick-up time, a drop-off location, a finaldestination of the user, one or more user preferences, user information,a number of riders for the requested ride service, and/or any other tripor user information.

At step 604, the method 600 can include receiving, at the client device(170), sensor data (e.g., 502) from one or more sensors (104) associatedwith the autonomous vehicle (102). As previously mentioned, the one ormore sensors can include, for example and without limitation, one ormore image sensors (e.g., camera sensors), LIDARs, radars, GPS sensors,motion detection sensors, speedometers, light detectors, audio sensors,braking sensors, ultrasonic sensors, position sensors, steering anglesensors, steering wheel rotation sensors, turn signal sensors, parkingsensors, gear position sensors, brake light sensors, odometers,proximity sensors, structured-light three-dimensional scanners, inertialmeasurement sensors (e.g., inertial measurement units, gyroscopes,accelerometers, etc.), temperature sensors, and/or any other sensor thatcan provide information used to determine or verify AV state, behavior,and/or context/environment information.

In some cases, the sensor data can include, for example and withoutlimitation, GPS data, inertial measurements (e.g., motion measurements,pose measurements, acceleration measurements, etc.), image data (e.g.,captured images or video depicting a scene/environment around theautonomous vehicle), radar measurements, LIDAR measurements or pointcloud information, position measurements (e.g., x, y, z coordinatesand/or pitch, roll, yaw measurements), etc. In some cases, the sensordata 502 can include other data such as cellular triangulation data,data indicating a location of a network (e.g., a WIFI network locationor Internet Protocol address) to which the client device (170) isconnected, data provided by the user, data provided by another user inthe autonomous vehicle, Internet data, traffic data, weather data, newsreport data, etc.

In some examples, the client device (170) can receive other informationfrom the autonomous vehicle (e.g., via the internal computing system 110and/or the remote computing system 150) in addition to the sensor data.For example, the client device (170) can receive virtual content tooverlay on an interface (e.g., 200, 300, 400) or an interface rendering(e.g., map 202, map 302, video feed 402, information section 212,information section 312, information section 404, etc.), messages todisplay on the client device (170), notifications or alerts to displayon the client device (170), images, graphical content, descriptiveinformation, etc.

At step 606, the method 600 can include determining, based on the sensordata, a state and context of the autonomous vehicle while the autonomousvehicle is en route to the pick-up location.

At step 608, the method 600 can include presenting, at a displayinterface (e.g., 200, 300, 400), a map (e.g., 202, 302, 402) depictingone or more visual indicators (e.g., 220-228, 304-310, 320-326, 330-334,340, etc.) of the state and context of the autonomous vehicle. The stateand context of the autonomous vehicle can include, for example andwithout limitation, a location of the autonomous vehicle, a behavior ofthe autonomous vehicle, one or more operations or conditions of theautonomous vehicle, an environment associated with the autonomousvehicle, etc.

In some examples, the state and context of the autonomous vehicle caninclude one or more AV operations. The one or more AV operations caninclude, for example and without limitation, driving, reversing,parking, braking, turning, activating or deactivating (e.g., turning onor off, maintaining on or off, etc.) a turn signal, honking a horn,activating or deactivating a brake light, changing a gear (e.g.,switching gears), changing a velocity (e.g., increasing or decreasing avehicle speed), activating or deactivating windshield wipers, changing aroute, picking up another rider, stopping (e.g., at a traffic signal orbefore an object blocking a path), changing driving lanes, performing au-turn, activating or deactivating any vehicle controls, and/orperforming any other maneuver. In some implementations, the mappresented at the display interface can depict the autonomous vehicleperforming the one or more AV operations.

In further examples, the state and context of the autonomous vehicle caninclude a temperature in an interior (e.g., cabin 138) of the autonomousvehicle, an AV battery status, a velocity of the autonomous vehicle, abehavior of the autonomous vehicle, a path of the autonomous vehicle tothe pick-up location, an environment associated with the autonomousvehicle, one or more conditions in the environment associated with theautonomous vehicle, etc. In some examples, the one or more conditions inthe environment associated with the autonomous vehicle can include atraffic condition, a road closure, a traffic accident, a hazard, atraffic signal (e.g., a traffic light, a stop sign, a yield sign, atraffic sign, etc.), an event in the environment, an obstacle along thepath of the autonomous vehicle, etc.

Moreover, in some cases, the one or more visual indicators can identify,and/or provide a visualization of, the one or more AV operations, one ormore aspects of the one or more AV operations, and/or any otherportions/aspects of the state and context of the autonomous vehicle. Forexample, in some cases, the one or more visual indicators can identify,and/or provide a visualization of, a status of a turn signal (e.g.,active or inactive, blinking/flashing, etc.) on the autonomous vehicle,a status of the brake light (e.g., active or inactive,blinking/flashing, etc.) on the autonomous vehicle, a rotation of thesteering wheel on the autonomous vehicle, a current gear active at theautonomous vehicle, a gear change at the autonomous vehicle, a u-turnoperation, a lane change operation, a reversing operation, a parkingoperation, an acceleration or deceleration, etc.

In other examples, the one or more visual indicators can identify,and/or provide a visualization of, a temperature in the interior (e.g.,cabin 138) of the autonomous vehicle, the AV battery status, a velocityof the autonomous vehicle, a behavior of the autonomous vehicle, a pathof the autonomous vehicle to the pick-up location, one or moreconditions in the environment associated with the autonomous vehicle, astatus of the one or more conditions in the environment associated withthe autonomous vehicle, etc. As previously noted, in some examples, theone or more conditions in the environment can include an obstacle alongthe path of the autonomous vehicle and/or an event in the environment.

In some examples, the obstacle can include, without limitation, astopped vehicle, an animal, a pot hole, a person, emergency personnel,emergency equipment, a construction, a street fair, flooding, etc.Moreover, in some examples, the event in the environment associated withthe autonomous vehicle can include, without limitation, a sportingevent, a concert, a traffic signal malfunction, an outdoor event, anearby conference, a traffic event, etc. In some cases, the one or morevisual indicators provide a visual representation of the obstacle alongthe path of the autonomous vehicle, a visual representation of theevent, a visual representation of one or more circumstances associatedwith the event, etc. Also, in some cases, the map can depict one or moreaspects of the environment associated with the autonomous vehicle suchas, for example, a landscape, a view, a condition, an object, a setting,a situation or circumstance, a state, a context, etc.

In some cases, the method 600 can further include tracking, based onadditional sensor data from the one or more sensors associated with theautonomous vehicle, the state and context of the autonomous vehiclewhile the autonomous vehicle is en route to the pick-up location and/orwaiting for the user; and presenting, on the map, one or more updatedvisual indicators of an updated state and context of the autonomousvehicle. In some examples, the updated state and context of theautonomous vehicle can include, without limitation, an updated locationof the autonomous vehicle, an updated behavior of the autonomousvehicle, and one or more conditions in a current environment associatedwith the autonomous vehicle, one or more current operations of theautonomous vehicle, one or more current controls of the autonomousvehicle, one or more current conditions of the autonomous vehicle, etc.

In some cases, the method 600 can also include presenting, at thedisplay interface, an information section (e.g., 212, 312, 404) thatprovides a description of one or more aspects of the state and contextof the autonomous vehicle, trip information (e.g., statistics, duration,estimated time of arrival, conditions, route, events, etc.), one or morenotifications, a status of one or more autonomous vehicle controls(e.g., brake lights, turn signals, steering wheel, brakes, windows,doors, navigation controls, pedal, gears, mechanical control systems,etc.

As described herein, one aspect of the present technology includesgathering and using data available from various sources to improvequality and experience. The present disclosure contemplates that in someinstances, this gathered data may include personal information. Thepresent disclosure contemplates that the entities involved with suchpersonal information respect and value privacy policies and practices.

FIG. 7 illustrates an example computing system 700 which can be, forexample, any computing device making up internal computing system 110,remote computing system 150, a client device (e.g., 170) executingrideshare application 172, or any other computing device. In FIG. 7 ,the components of the computing system 700 are in communication witheach other using connection 705. Connection 705 can be a physicalconnection via a bus, or a direct connection into processor 710, such asin a chipset architecture. Connection 705 can also be a virtualconnection, networked connection, or logical connection.

In some implementations, computing system 700 is a distributed system inwhich the functions described in this disclosure can be distributedwithin a datacenter, multiple data centers, a peer network, etc. In someimplementations, one or more of the described system componentsrepresents many such components each performing some or all of thefunction for which the component is described. In some implementations,the components can be physical or virtual devices.

Example system 700 includes at least one processing unit (CPU orprocessor) 710 and connection 705 that couples various system componentsincluding system memory 715, such as read-only memory (ROM) 720 andrandom access memory (RAM) 725 to processor 710. Computing system 700can include a cache 712 of high-speed memory connected directly with, inclose proximity to, or integrated as part of processor 710.

Processor 710 can include any general purpose processor and a hardwareservice or software service, such as services 732, 734, and 736 storedin storage device 730, configured to control processor 710 as well as aspecial-purpose processor where software instructions are incorporatedinto the actual processor design. Processor 710 may essentially be acompletely self-contained computing system, containing multiple cores orprocessors, a bus, memory controller, cache, etc. A multi-core processormay be symmetric or asymmetric.

To enable user interaction, computing system 700 includes an inputdevice 745, which can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech, etc. Computingsystem 700 can also include output device 735, which can be one or moreof a number of output mechanisms known to those of skill in the art. Insome instances, multimodal systems can enable a user to provide multipletypes of input/output to communicate with computing system 700.Computing system 700 can include communications interface 740, which cangenerally govern and manage the user input and system output. There isno restriction on operating on any particular hardware arrangement, andtherefore the basic features here may easily be substituted for improvedhardware or firmware arrangements as they are developed.

Storage device 730 can be a non-volatile memory device and can be a harddisk or other types of computer readable media which can store data thatare accessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs), read-only memory (ROM), and/or somecombination of these devices.

The storage device 730 can include software services, servers, services,etc., that when the code that defines such software is executed by theprocessor 710, it causes the system to perform a function. In someimplementations, a hardware service that performs a particular functioncan include the software component stored in a computer-readable mediumin connection with the necessary hardware components, such as processor710, connection 705, output device 735, etc., to carry out the function.

For clarity of explanation, in some instances, the present technologymay be presented as including individual functional blocks includingfunctional blocks comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

Any of the steps, operations, functions, or processes described hereinmay be performed or implemented by a combination of hardware andsoftware services or services, alone or in combination with otherdevices. In some embodiments, a service can be software that resides inmemory of a client device and/or one or more servers of a contentmanagement system and perform one or more functions when a processorexecutes the software associated with the service. In some embodiments,a service is a program or a collection of programs that carry out aspecific function. In some embodiments, a service can be considered aserver. The memory can be a non-transitory computer-readable medium.

In some implementations, the computer-readable storage devices, mediums,and memories can include a cable or wireless signal containing a bitstream and the like. However, when mentioned, non-transitorycomputer-readable storage media expressly exclude media such as energy,carrier signals, electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer-readable media. Such instructions can comprise,for example, instructions and data which cause or otherwise configure ageneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The executable computer instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, solid-state memory devices, flash memory, USB devices providedwith non-volatile memory, networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include servers,laptops, smartphones, small form factor personal computers, personaldigital assistants, and so on. The functionality described herein alsocan be embodied in peripherals or add-in cards. Such functionality canalso be implemented on a circuit board among different chips ordifferent processes executing in a single device, by way of furtherexample.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims.

Claim language reciting “at least one of” a set indicates that onemember of the set or multiple members of the set satisfy the claim. Forexample, claim language reciting “at least one of A and B” means A, B,or A and B.

What is claimed is:
 1. A method comprising: receiving a user requestfrom a user device for a ride from an autonomous vehicle (AV), whereinthe user request specifies a pick-up location associated with a user;receiving a video feed captured by one or more camera sensors on the AV,the video feed depicting a local scene associated with the AV;identifying a parking location for the user to enter the AV; andproviding, to the user device, the video feed and a message identifyingthe parking location for the user to enter the AV within the pick-uplocation, wherein the user device concurrently displays the video feedand the message.
 2. The method of claim 1, wherein one or more AVoperations comprise at least one of driving, reversing, parking,braking, turning, activating a turn signal, activating a brake light,changing a gear, changing a velocity of the AV, and performing a u-turn.3. The method of claim 1, the method further comprising: presenting, ata display interface, a map depicting one or more visual indicators of astate and context of the AV, wherein the map depicts the AV performingone or more AV operations, and wherein one or more visual indicatorsidentify at least one of a first status of a turn signal on the AV, asecond status of a brake light on the AV, a rotation of a steering wheelon the AV, a current gear active at the AV, a gear change at the AV, anda u-turn operation.
 4. The method of claim 3, wherein the state andcontext of the AV further comprises at least one of a temperature in aninterior of the AV, an AV battery status, a velocity of the AV, abehavior of the AV, a path of the AV to the pick-up location, anenvironment associated with the AV, and one or more conditions in theenvironment associated with the AV, wherein the one or more conditionsin the environment associated with the AV comprise at least one oftraffic, a road closure, a traffic accident, a hazard, a traffic signal,an event in the environment associated with the AV, and an obstaclealong the path of the AV.
 5. The method of claim 4, wherein the mapdepicts one or more aspects of the environment associated with the AV,the method further comprising: presenting, at a display interface of theuser device, an information section providing at least one of adescription of one or more aspects of a state and context of the AV,trip information, one or more notifications, and a status of one or moreAV controls.
 6. The method of claim 3, wherein the one or more visualindicators identify at least one of a temperature in an interior of theAV, an AV battery status, a velocity of the AV, a behavior of the AV, apath of the AV to the pick-up location, one or more conditions in anenvironment associated with the AV, and a status of the one or moreconditions in the environment associated with the AV.
 7. The method ofclaim 1, wherein an obstacle along a path of the AV comprises at leastone of a stopped vehicle, an animal, a pot hole, a person, emergencypersonnel, emergency equipment, a construction zone, a street fair, andflooding, wherein one or more visual indicators provide a visualrepresentation of the obstacle along the path of the AV and one or morecircumstances associated with an event in an environment.
 8. A systemcomprising: one or more sensors of an autonomous vehicle; and aprocessor for executing instructions stored in memory, wherein executionof the instructions by the processor executes: receiving a user requestfrom a user device for a ride from an autonomous vehicle (AV), whereinthe user request specifies a pick-up location associated with a user;receiving a video feed captured by one or more camera sensors on the AV,the video feed depicting a local scene associated with the AV;identifying a parking location for the user to enter the AV; andproviding, to the user device, the video feed and a message identifyingthe parking location for the user to enter the AV within the pick-uplocation, wherein the user device concurrently displays the video feedand the message.
 9. The system of claim 8, wherein one or more AVoperations comprise at least one of driving, reversing, parking,braking, turning, activating a turn signal, activating a brake light,changing a gear, changing a velocity of the AV, and performing a u-turn.10. The system of claim 8, wherein execution of the instructions by theprocessor further executes: presenting, at a display interface, a mapdepicting one or more visual indicators of a state and context of theAV, wherein the map depicts the AV performing one or more AV operations,and wherein one or more visual indicators identify at least one of afirst status of a turn signal on the AV, a second status of a brakelight on the AV, a rotation of a steering wheel on the AV, a currentgear active at the AV, a gear change at the AV, and a u-turn operation.11. The system of claim 10, wherein the state and context of the AVfurther comprises at least one of a temperature in an interior of theAV, an AV battery status, a velocity of the AV, a behavior of the AV, apath of the AV to the pick-up location, an environment associated withthe AV, and one or more conditions in the environment associated withthe AV, wherein the one or more conditions in the environment associatedwith the AV comprise at least one of traffic, a road closure, a trafficaccident, a hazard, a traffic signal, an event in the environmentassociated with the AV, and an obstacle along the path of the AV. 12.The system of claim 11, wherein the map depicts one or more aspects ofthe environment associated with the AV, wherein execution of theinstructions by the processor further executes: presenting, at a displayinterface of the user device, an information section providing at leastone of a description of one or more aspects of a state and context ofthe AV, trip information, one or more notifications, and a status of oneor more AV controls.
 13. The system of claim 10, wherein the one or morevisual indicators identify at least one of a temperature in an interiorof the AV, an AV battery status, a velocity of the AV, a behavior of theAV, a path of the AV to the pick-up location, one or more conditions inan environment associated with the AV, and a status of the one or moreconditions in the environment associated with the AV.
 14. The system ofclaim 8, wherein an obstacle along a path of the AV comprises at leastone of a stopped vehicle, an animal, a pot hole, a person, emergencypersonnel, emergency equipment, a construction zone, a street fair, andflooding, wherein one or more visual indicators provide a visualrepresentation of the obstacle along the path of the AV and one or morecircumstances associated with an event in an environment.
 15. Anon-transitory computer readable medium comprising instructions, theinstructions, when executed by a computing system, cause the computingsystem to: receive a user request from a user device for a ride from anautonomous vehicle (AV), wherein the user request specifies a pick-uplocation associated with a user; receive a video feed captured by one ormore camera sensors on the AV, the video feed depicting a local sceneassociated with the AV; identify a parking location for the user toenter the AV; and provide, to the user device, the video feed and amessage identifying the parking location for the user to enter the AVwithin the pick-up location, wherein the user device concurrentlydisplays the video feed and the message.
 16. The non-transitory computerreadable medium of claim 15, wherein one or more AV operations compriseat least one of driving, reversing, parking, braking, turning,activating a turn signal, activating a brake light, changing a gear,changing a velocity of the AV, and performing a u-turn.
 17. Thenon-transitory computer readable medium of claim 15, the instructionsfurther causing the computing system to: present, at a displayinterface, a map depicting one or more visual indicators of a state andcontext of the AV, wherein the map depicts the AV performing one or moreAV operations, and wherein one or more visual indicators identify atleast one of a first status of a turn signal on the AV, a second statusof a brake light on the AV, a rotation of a steering wheel on the AV, acurrent gear active at the AV, a gear change at the AV, and a u-turnoperation.
 18. The non-transitory computer readable medium of claim 17,wherein the state and context of the AV further comprises at least oneof a temperature in an interior of the AV, an AV battery status, avelocity of the AV, a behavior of the AV, a path of the AV to thepick-up location, an environment associated with the AV, and one or moreconditions in the environment associated with the AV, wherein the one ormore conditions in the environment associated with the AV comprise atleast one of traffic, a road closure, a traffic accident, a hazard, atraffic signal, an event in the environment associated with the AV, andan obstacle along the path of the AV.
 19. The non-transitory computerreadable medium of claim 17, wherein the one or more visual indicatorsidentify at least one of a temperature in an interior of the AV, an AVbattery status, a velocity of the AV, a behavior of the AV, a path ofthe AV to the pick-up location, one or more conditions in an environmentassociated with the AV, and a status of the one or more conditions inthe environment associated with the AV.
 20. The non-transitory computerreadable medium of claim 15, wherein an obstacle along a path of the AVcomprises at least one of a stopped vehicle, an animal, a pot hole, aperson, emergency personnel, emergency equipment, a construction zone, astreet fair, and flooding, wherein one or more visual indicators providea visual representation of the obstacle along the path of the AV and oneor more circumstances associated with an event in an environment.